2113 nimrod regrid by BernardClaxton · Pull Request #2269 · MetOffice/CSET

BernardClaxton (BernardClaxton) · 2026-07-14T13:47:37Z

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

Githib copilot was used to help write some of the tests in test_radar_filter.py.

Documentation has been updated to reflect change.
New code has tests, and affected old tests have been updated.
All tests and CI checks pass.
Ensured the pull request title is descriptive.
Attributed any Generative AI, such as GitHub Copilot, used in this PR.
Marked the PR as ready to review.

github-actions · 2026-07-14T13:50:54Z

Total coverage: 93% (HTML report)

Name                                                              Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------------------------
src/CSET/__init__.py                                                113      4     14      0    97%
src/CSET/_common.py                                                 156      0     54      0   100%
src/CSET/cset_workflow/app/fetch_fcst/bin/fetch_data.py             117     28     26      0    78%
src/CSET/cset_workflow/app/fetch_nimrod/bin/fetch_nimrod.py          85      8     38     16    80%
src/CSET/cset_workflow/app/finish_website/bin/finish_website.py      79      2      8      2    95%
src/CSET/cset_workflow/app/parbake_recipes/bin/parbake.py            29      0      8      0   100%
src/CSET/cset_workflow/app/send_email/bin/send_email.py              25      0      4      0   100%
src/CSET/cset_workflow/lib/python/jinja_utils.py                     17      0      6      0   100%
src/CSET/extract_workflow.py                                        105      1     26      1    98%
src/CSET/graph.py                                                    44      0     14      0   100%
src/CSET/operators/__init__.py                                       89      0     26      0   100%
src/CSET/operators/_atmospheric_constants.py                          9      0      0      0   100%
src/CSET/operators/_colormaps.py                                    250      3     72      4    98%
src/CSET/operators/_stash_to_lfric.py                                 3      0      0      0   100%
src/CSET/operators/_utils.py                                        200      8     74      6    95%
src/CSET/operators/ageofair.py                                      142      7     64      5    94%
src/CSET/operators/aggregate.py                                     184      8     60      7    94%
src/CSET/operators/aviation.py                                       61      0     18      0   100%
src/CSET/operators/collapse.py                                      162      8     80      3    94%
src/CSET/operators/constraints.py                                   115      7     50      2    93%
src/CSET/operators/convection.py                                     38      4     10      2    88%
src/CSET/operators/ensembles.py                                      27      0     14      0   100%
src/CSET/operators/feature.py                                        44      0     10      0   100%
src/CSET/operators/filters.py                                        67      2     30      0    98%
src/CSET/operators/fluxes.py                                         41      0     10      0   100%
src/CSET/operators/humidity.py                                      135      0     52      0   100%
src/CSET/operators/imageprocessing.py                                57      0     16      0   100%
src/CSET/operators/mesoscale.py                                      18      0      2      0   100%
src/CSET/operators/misc.py                                          172      1     72      3    98%
src/CSET/operators/plot.py                                         1174    156    440     71    83%
src/CSET/operators/power_spectrum.py                                156     10     60      8    91%
src/CSET/operators/precipitation.py                                 204      2     92      2    99%
src/CSET/operators/pressure.py                                       41      0     12      0   100%
src/CSET/operators/radar_filter.py                                  111      4     44      4    95%
src/CSET/operators/read.py                                          453     25    200     15    94%
src/CSET/operators/regrid.py                                        147      1     70      3    98%
src/CSET/operators/scoreswrappers.py                                226     22     74      6    89%
src/CSET/operators/temperature.py                                   121      0     32      0   100%
src/CSET/operators/transect.py                                       63      0     24      0   100%
src/CSET/operators/wind.py                                           46      3     10      2    91%
src/CSET/operators/write.py                                          15      0      6      0   100%
src/CSET/recipes/__init__.py                                        104      0     28      0   100%
src/CSET/sample_data/__init__.py                                      0      0      0      0   100%
---------------------------------------------------------------------------------------------------
TOTAL                                                              5445    314   1950    162    93%

…lations #2113

…, np.nanmax, np.nanmean #2113

…2113

… to NaN #2113

Sylvia Bohnenstengel (Sylviabohnenstengel) · 2026-10-14T17:46:09Z

+title=Produce Nimrod 2D surface plots
+description=Produce 2D surface plots of radar observations.
+help=Switch to produce 2D surface plots utilising Nimrod radar observations.
+     Plots are produced of the 2D surface rainfall for each of the selected Nimrod sources.


more precision needed: Nimrod provides rainfall amounts

Sylvia Bohnenstengel (Sylviabohnenstengel) · 2026-10-14T17:47:27Z

+title=Produce Nimrod radar histograms
+description=Produce histograms incorporating radar observations.
+help=Switch to produce histograms utilising Nimrod radar observations.
+     The histograms will combine both model rainfall and radar observations.


This will likely change when lfric gives us accumulations as well, but worth being precise her that Nimrod gives as amounts as an accumulation for sum products and a 5min "snapshot" for the other product, whereas the models currently gives us rainfall_rate i.e. amount per time as a snapshot

Sylvia Bohnenstengel (Sylviabohnenstengel) · 2026-10-14T17:47:59Z

+def select_radar_source(input_source_list):
+    """Select the preferred radar source from a list of sources."""
+    # Preference list for Nimrod sources, with most preferred heading the list.
+    nimrod_preference = ["Nimrod2km", "Nimrodxkm", "Nimrod1km"]


needs explanation for xkm

Sylvia Bohnenstengel (Sylviabohnenstengel) · 2026-10-15T12:51:07Z

                # for this is set in v["weights"].
                if v["weights"] == "True" and nimrod_weights_exist == "True":
                    filename_wei_nc = (
                        f"{nimrod_dir_wei}/{date_use.strftime('%Y%m%d%H%M')}"


Now the radar data are written out with a timepoint in the filename that is shifted by 1hour for the radar_offset. So a file with CYCLE_POINT timestamp in the name is now renamed to CYCLE_POINT+1 in the name while the time coordinates inside have not changed. Is this correct given that the original filename has the endtime in the filename?

Need to check the filenames written to disk: Do we now have CYCLE_POINT+1 as filename, but time bounds in the file with [CYCLE_POINT-1:CYCLE_POINT]?

Carol Halliwell (cehalliwell)

I have has a look through and left comments/suggestions.

Could you add a description of what this PR is doing?

Have you tested all the different combinations (or as many as possible) of radar type/model?

Carol Halliwell (cehalliwell) · 2026-10-13T13:43:11Z

    ]

+    # Specify the boundary margin width to use when trimming model fields.
+    boundary_margin = 16


Does this need to be fixed? If not, can it be set by the user?

Carol Halliwell (cehalliwell) · 2026-10-13T13:53:27Z

+description=Produce histograms incorporating radar observations.
+help=Switch to produce histograms utilising Nimrod radar observations.
+     The histograms will combine both model rainfall and radar observations.
+     The model rainfall variable is selected in the Rose tab Diagnostics/Surface (2D) fields.


Is this correct? Other code suggests rainfall variable is fixed, i.e.
"MODEL_VARNAME": "surface_microphysical_rainfall_rate",

James Ross (james-a-f-ross) · 2026-10-15T14:48:15Z

do the recipes maybe belong in their own directory now? e.g recipes/radar or recipes/surface_fields/radar?

Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>

…plot.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

…togram_domain_mean.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

…togram_sequence.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

James Ross (james-a-f-ross) · 2026-10-16T14:34:29Z

I have started to make some changes in this branch, some thing I'm not too sure about though.

James Warner (jwarner8) · 2026-10-21T13:50:38Z

I've been doing some testing with this branch on Sana's sensitivity experiments over the UK. I had to change the recipe arguments of radar_filter.radar_mask_loop and radar_filter.radar_mask from outputs to required_outputs to get some of the recipes to work (as the operators in radar_filter.py expect required_outputs as the named argument).

The spatial maps of radar seem to work. Model comparisons don't seem to work. Errors I get include;

histogram_surface_microphysical_rainfall_rate_case_study_common_domain_model_and_nimrod_radar and histogram_surface_microphysical_rainfall_rate_sequence_plots_common_domain_model_and_nimrod_radar recipes, produce
Object {obj} cannot be put in a cubelist, as it is not a Cube., traceback to

CSET/src/CSET/operators/radar_filter.py", line 401, in radar_mask_loop filtered_cubes.append(filtered_model)

One of the cycles failed from 0221111T0000Z/data/Nimrod5min 'no files found'. I guess we might have just been unlucky with that date if there are no radar obs.
The timeseries don't seem to include the radar (domain average)? All other models are there. Are domain averages not supported yet?
As above, assume aggregated histograms not supported, as this succeeds and produces model histograms, but no radar on this?
Some of the bake tasks are very slow (80 minutes to run), and appear to only be running one task at the end for most of it. Is part of the regridding that expensive and is this expected behaviour?

James Ross (james-a-f-ross) · 2026-10-22T10:50:33Z

+            if nimrod_field == "Nimrod_comp_xkm":
+                radar_offset = timedelta(hours=1.0)
+            if nimrod_field == "Nimrod_comp_1km":
+                radar_offset = timedelta(hours=1.0)
+            if nimrod_field == "Nimrod_comp_2km":
+                radar_offset = timedelta(hours=1.0)


could do this witth a Match Case statement too and I feel it's a bit cleaner.

BernardClaxton (BernardClaxton) self-assigned this Jul 14, 2026

Base automatically changed from 1759_import_nimrod to main July 16, 2026 15:39

BernardClaxton (BernardClaxton) mentioned this pull request Jul 20, 2026

New colormaps test to cover Nimrod wts case #2302

Merged

4 of 6 tasks

BernardClaxton (BernardClaxton) added 12 commits July 20, 2026 11:03

Recipe for histograms of masked Nimrod accumulations #2113

0b52c83

Loader for Nimrod masked histograms #2113

391c433

Function to produce Nimrod masked cubes #2113

8a091ec

Recipe to produce mean time series of masked Nimrod accumulations #2113

a1fc575

Loader for recipe to produce mean time series of masked Nimrod accumu…

4cfbdfd

…lations #2113

Nimrod weights 2-d plots to use gray for out of radar range domain #2113

fa0f7cd

jaml recipe for plotting masked model rainfall #2113

d6d441b

function radar_apply_mask in radar_filter.py #2113

93825e2

radar_apply_mask now uses function apply_mask in filters.py #2113

99cc537

loaders for radar_mask_model.yaml #2113

3268736

for 2-d surface plots replaced np.min, np.max, np.mean with np.nanmin…

33a20f2

…, np.nanmax, np.nanmean #2113

Grey out domain beyond radar range for rainfall masked spatial plots #…

4b81b55

…2113

BernardClaxton (BernardClaxton) force-pushed the 2113_nimrod_regrid branch from 60bea6d to 4b81b55 Compare July 20, 2026 10:54

BernardClaxton (BernardClaxton) linked an issue Jul 20, 2026 that may be closed by this pull request

Nimrod regridding and filtering #2113

Open

When regridding model using nimrod wts field, set extrapolated values…

493ddfa

… to NaN #2113

Sylvia Bohnenstengel (Sylviabohnenstengel) self-requested a review July 23, 2026 10:14

BernardClaxton (BernardClaxton) added 10 commits August 27, 2026 09:38

Radar operators in radar_filter.py #2113

79d9476

Radar loaders #2113

da7576d

Radar recipe for domain mean histogram #2113

2d973f2

Radar recipe for histogram sequences #2113

a9e0d13

Radar loaders #2113

6073ff9

recipe histogram mean #2113

7acc815

recipe 2-d radar plots #2113

36dd595

recipe radar_dev3 plots #2113

ae5ba6c

recipe radar_masked_mean_time_series #2113

596cace

recipe radar_masked_time_series #2113

b4390e1

Carol Halliwell (cehalliwell) reviewed Oct 13, 2026

View reviewed changes

Comment thread src/CSET/recipes/surface_fields/nimrod/radar_common_domain_2d_plot.yaml Outdated

Carol Halliwell (cehalliwell) reviewed Oct 13, 2026

View reviewed changes

Comment thread src/CSET/recipes/surface_fields/nimrod/radar_common_domain_histogram_domain_mean.yaml Outdated

Sylvia Bohnenstengel (Sylviabohnenstengel) requested changes Oct 15, 2026

View reviewed changes

Sylvia Bohnenstengel (Sylviabohnenstengel) self-requested a review October 15, 2026 12:54

Carol Halliwell (cehalliwell) reviewed Oct 15, 2026

View reviewed changes

James Ross (james-a-f-ross) and others added 6 commits October 16, 2026 10:05

Apply suggestion from @Sylviabohnenstengel

86d2784

Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>

Update src/CSET/operators/radar_filter.py

51fb232

Update src/CSET/loaders/radar.py

9b4dcaa

Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>

Update src/CSET/operators/radar_filter.py

b753505

type hinting

d7b621c

outputs -> required_outputs. match statement instead of if statements

3bfbe9c

James Ross (james-a-f-ross) reviewed Oct 16, 2026

View reviewed changes

Comment thread src/CSET/operators/radar_filter.py Outdated

James Ross (james-a-f-ross) and others added 11 commits October 16, 2026 10:40

simiplify radar_mask_loop

1a0bcb4

fix unit tests

d381a8e

Update src/CSET/loaders/radar.py

465fc5b

Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>

combine conditional

d491106

fix uneven boudnary

660b776

make mask boundary seperate function and unit test it

e597460

move radar recipes to own folder

65ed6d7

removing some redundant code

b2642c3

Update src/CSET/recipes/surface_fields/nimrod/radar_common_domain_2d_…

df5aace

…plot.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

Update src/CSET/recipes/surface_fields/nimrod/radar_common_domain_his…

fc74417

…togram_domain_mean.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

Update src/CSET/recipes/surface_fields/nimrod/radar_common_domain_his…

245f356

…togram_sequence.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>

James Ross (james-a-f-ross) added 2 commits October 20, 2026 14:30

code cleanup

59c9546

fix test

66e37df

outputs -> required_outputs

bce263e

James Ross (james-a-f-ross) reviewed Oct 22, 2026

View reviewed changes

Conversation

BernardClaxton (BernardClaxton) commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution checklist

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Carol Halliwell (cehalliwell) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

James Ross (james-a-f-ross) commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

James Ross (james-a-f-ross) commented Oct 16, 2026

Uh oh!

James Warner (jwarner8) commented Oct 21, 2026 • edited by james-a-f-ross Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

BernardClaxton (BernardClaxton) commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

James Ross (james-a-f-ross) commented Oct 15, 2026 •

edited

Loading

James Warner (jwarner8) commented Oct 21, 2026 •

edited by james-a-f-ross

Loading